Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

Also, I think which would be awesome to have would be to add some tests in maskrcnn-benchmark/tests, which we are currently lacking and would make reviewing the changes much easier!

@fmassa I think it should be consistent with Detectron now.

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

I'll have another closer look today, but would you mind adding some tests in tests/? It will make things much easier!

@fmassa I'm not familiar with how to write the tests.

Ok, no worries

@fmassa I've added some tests about segmentation_mask. However, the transpose and resize can not pass the test. Some help is needed.

For resize I can understand there maybe some numerical difference because of the interpolation, but I don't know how to make them equivalent. For transpose the difference is much stranger, I don't know where the difference is.

For the resize, I'd check if the difference appears in the boundaries. Also, the values should be 0-1.
For the transpose, I'd see if the difference disappears if you set this TO_REMOVE = 0. That could explain the difference.

Also, visualizing both results in the image space is going to be very helpful.

@fmassa For transpose, I changed TO_REMOVE=0 in class Polygons. The difference is smaller but still exists. For resize, I've visualized the result, the difference almost comes from the boundaries.

Ok, this is good progress, thanks!

I'd need to look more closely to see where the difference might come from for the transpose, there might be a few pixels off and this can be observed also by viewing the images.

For resize, I'd need to check a bit more carefully to see if I spot anything, but I might not have the time today nor tomorrow.

@fmassa For transpose, I changed TO_REMOVE=0 in class Polygons. The difference is smaller but still exists. For resize, I've visualized the result, the difference almost comes from the boundaries.

If I use the segmentation_mask.py you wrote, are there other changes I should apply on other files?
It looks like the functions and ports fit the original code well.
What's more, are there other things I should take attention? Maybe I can try the transforms without transposing first? >< Thank you!

@JoyHuYY1412 you'd need to check the interpolation (to use bilinear instead of nearest), apart from that, the rest should be unchanged (or almost).

@txytju do you mind finishing this PR, given that you managed to make it work for your case?

@JoyHuYY1412 you'd need to check the interpolation (to use bilinear instead of nearest), apart from that, the rest should be unchanged (or almost).
Thank you~

I like this!

I faced a problem with cropped_mask = self.mask[box[1]: box[3], box[0]: box[2]] in the part of the code below where I had box[0] and box[2] equal, resulting in width=0, causing an error.

def crop(self, box):
        box = [int(b) for b in box]
        w, h = box[2] - box[0], box[3] - box[1]
        w = max(w, 1)
        h = max(h, 1)
        cropped_mask = self.mask[box[1]: box[3], box[0]: box[2]]
        return Mask(cropped_mask, size=(w, h), mode=self.mode)

This happened because I called cropped_mask = segmentation_mask.crop(proposal) where proposal is tensor([610.0664, 258.8555, 610.7168, 269.9121]) which got rounded to tensor([610, 258, 610, 269])

@IssamLaradji I think it should be w, h = box[2] - box[0] + 1, box[3] - box[1] + 1.

And I guess round is more suitable than int?

@botcs if you could write unit tests for this PR, it would be awesome!

using the available test_segmentation_mask, I have tried to visualize the Polygon and the Mask tests, and here is a notebook of my observations:
https://gist.github.com/botcs/95176d877dcd26e48e46cceecdac5763

@botcs that's awesome, thanks for the notebook!
So indeed we have the boundary effects as before.

One last thing I'd like to do to verify if this boundary effects actually matter is to run a training with https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/configs/quick_schedules/e2e_mask_rcnn_R_50_FPN_quick.yaml , which is a fast training and should take ~10 min to run, using the Mask class instead of Polygons.

For that, I'd (locally, this is not to be commited) modify

I'd expect to have as results something like

EXPECTED_RESULTS: [['coco_2014_minival', 'box', 'AP', [0.082300, 0.001682]], ['coco_2014_minival', 'mask', 'AP', [0.075039, 0.001872]]]

, so box AP should be around 8.2, and mask AP around 7.5

Could you do that?

Thanks!

@botcs that's awesome, thanks for the notebook!
So indeed we have the boundary effects as before.

One last thing I'd like to do to verify if this boundary effects actually matter is to run a training with https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/configs/quick_schedules/e2e_mask_rcnn_R_50_FPN_quick.yaml , which is a fast training and should take ~10 min to run, using the Mask class instead of Polygons.

For that, I'd (locally, this is not to be commited) modify

maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/coco.py

Lines 82 to 84 in f8b0118
masks = [obj["segmentation"] for obj in anno]
masks = SegmentationMask(masks, img.size)
target.add_field("masks", masks)

so that it uses Mask instead.

I'd expect to have as results something like

EXPECTED_RESULTS: [['coco_2014_minival', 'box', 'AP', [0.082300, 0.001682]], ['coco_2014_minival', 'mask', 'AP', [0.075039, 0.001872]]]

, so box AP should be around 8.2, and mask AP around 7.5

Could you do that?

Thanks!

I have trained the model, which went fine, but the evaluation has failed with the following error:

  File "/home/csbotos/anaconda3/envs/debugmask/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/structures/segmentation_mask.py", line 206, in __init__
    if not isinstance(segms[0], (list, Polygons)):
IndexError: list index out of range

the whole output can be foun at this gist

I have suppressed the error by using single empty instance with an all-zero mask when the provided segms is an empty list. And the results are the following: box AP is 6.0 and mask AP is 5.9

It is quite below the expected performance, and I now rerun the training to see if it is still the case.

So I was curious about the expected results in @fmassa 's comment if they were correct, and I have ran the training using Polygons and Mask and the following results came back, two runs each:

• Polygon box AP is 4.4 and mask AP is 4.3
• Mask box AP is 6.0 and mask AP is 5.9

Which means that the Mask is doing better than the Polygon and the expected results for the training script were higher: bbox 8.2, mask 7.5

@IssamLaradji I think it should be w, h = box[2] - box[0] + 1, box[3] - box[1] + 1.

And I guess round is more suitable than int?

In the code, should be round(float(b)) ?

Hi guys,

A few days ago @fmassa mentioned in one of his comments the following:

In general, I think both box_list and segmentation_mask could benefit from some better design / cleanup, but I'm not sure what that would be

So I tried to reshape things a bit to accommodate better the different requirements, but it has radically changed a few concepts. I would be keen to learn about your opinions on it: #473

I am getting this with this code,

  File "/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/mask_head/loss.py", line 39, in project_masks_on_boxes 
    scaled_mask = cropped_mask.resize((M, M)) 
  File "/mnt/home/issam/Research_Ground/domain_adaptation/ann_utils.py", line 282, in resize 
    self.mask[None, None, :, :], (height, width), mode="bilinear" 
  File "/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 2447, in interpolate 
    return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners) 
RuntimeError: invalid argument 2: input and output sizes should be greater than 0, but got input (H: 2, W: 0) output (H: 28, W: 28) at /pytorch/aten/src/THNN/generic/SpatialUpSamplingBilinear.c:19 
Uncaught exception. Entering post mortem debugging 
Running 'cont' or 'step' will restart the program 

self.mask = tensor([], size=(2, 0))

@IssamLaradji does this also happen with #473 ?

@IssamLaradji I think it should be box = [max(round(float(b)), 0) for b in box]. The bad input size comes from a crop when a box coordinate somehow becomes -1.

Thanks for the initial work on this PR @wangg12 ! This has been merged in #473

Does this work yet for RLE? I am getting this when I pass a list of RLEs.

File "/maskrcnn-benchmark/maskrcnn_benchmark/structures/segmentation_mask.py", line 73, in __init__
    if len(masks.shape) == 2:
AttributeError: 'list' object has no attribute 'shape'

AttributeError: 'list' object has no attribute 'shape'
I met the same problem!

Hi @ShihuaiXu ,
Please visit the updates here